Telepresence using panoramic imaging and directional sound and motion

ABSTRACT

An imaging system comprising a panoramic visual image display, an associated directional sound playback device, and an associated motion reproduction device is disclosed. The imaging system conveys visual, sound and motion information related to a particular viewing direction to provide a realistic experience for the viewer. The imaging system can also comprise a panoramic visual image recording device capable of recording panoramic images, an associated directional sound capturing device capable of recording sound, and an associated directional motion capturing device capable of recording motion. Recorded panoramic images, sound and motion can be synchronously recorded to a common time code for simultaneous playback.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.60/592,533 filed Jul. 30, 2004, which is incorporated herein byreference.

FIELD OF THE INVENTION

The present invention relates to panoramic imaging, and moreparticularly relates to the use of panoramic visual images incombination with directional sound and motion to provide an immersiveimaging experience.

BACKGROUND INFORMATION

Panoramic imagery is able to capture a large azimuth view with asignificant elevation angle. In some cases, the view is achieved throughthe use of wide angle optics such as fish-eye lens. This view may beexpanded by combining or “stitching” a series of images from one or morecameras with overlapping fields of view into one continuous view. Inother cases it is achieved through the use of a combination of mirrorsand lens. Alternatively, the view may be developed by rotating animaging sensor so as to achieve a panorama. The panoramic view can becomposed of still images or, in cases where the images are taken at highfrequencies, the sequence can be interpreted as animation. Wide anglesassociated with panoramic imagery can cause the image to appear warped,i.e., the image does not correspond to a natural human view. Thisimagery can be unwarped by various means including software to display anatural view.

While systems have been proposed in which panoramic images can becreated in computer generated environments, such as withthree-dimensional models, a need remains for an imaging system in whichstill or video panoramic imagery is combined with directional sound anddirectional motion to provide realistic telepresence.

SUMMARY OF THE INVENTION

The present imaging system conveys not only visual information but alsoaudio information and motion information related to a particular viewingdirection to improve the realism to the viewer on playback. The imagingsystem comprises a panoramic visual image display, an associateddirectional sound playback device, and an associated motion reproductiondevice. The imaging system can also comprise a panoramic visual imagerecording device capable of recording panoramic images, an associateddirectional sound capturing device capable of recording sound, and anassociated directional motion capturing device capable of recordingmotion.

An aspect of the present invention is to provide an apparatus comprisinga panoramic visual image capturing device capable of recording panoramicvisual images, a sound capturing device capable of recording soundassociated with viewing directions for the visual images, and a motioncapturing device capable of recording motion associated with viewingdirections for the visual images.

Another aspect of the present invention is to provide a methodcomprising the steps of recording panoramic visual images, recordingsound associated with viewing directions for the visual images, andrecording motion associated with viewing directions for the visualimages.

Another aspect of the present invention is to provide an image playbacksystem comprising a panoramic visual image display for displayingpanoramic visual images, a sound playback device for reproducing soundassociated with viewing directions for the panoramic visual images, anda motion reproduction device for reproducing motion associated withviewing directions for the panoramic visual images, wherein thepanoramic visual images, sound and motion are synchronously reproduced.

A further aspect of the present invention is to provide an imageplayback method comprising the steps of displaying panoramic visualimages, reproducing sound associated with viewing directions for thepanoramic visual images, and reproducing motion associated with viewingdirections for the panoramic visual images, wherein the panoramic visualimages, sound and motion are synchronously reproduced.

These and other aspects of the present invention will be more apparentfrom the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a system for producingpanoramic images in accordance with the invention.

FIG. 2 is a raw image from a panospheric camera in accordance with theinvention.

FIG. 3 is the image from FIG. 2 displayed as a rectangular image using aprojection onto a cylindrical surface in accordance with the invention.

FIG. 4A is a schematic illustration of an imaging system comprising apanoramic camera with multiple microphones which are used to recordsound from one or more sound sources and a gyroscopic motion sensorwhich is used to record motion in one or more directions in accordancewith the invention.

FIG. 4B is a schematic illustration of an image playback systemcomprising a panoramic visual image display, an associated soundplayback device, and an associated motion reproduction device inaccordance with the invention.

FIG. 4C is a schematic illustration of an image playback systemcomprising a panoramic visual image display, an associated soundplayback device, and an associated motion reproduction device inaccordance with the invention.

FIG. 5 is a schematic illustration of the strength of a sound from aparticular sound source with respect to the current viewing direction inaccordance with the invention.

FIG. 6 is the rectangular projected image of FIG. 3, illustrating anangle between the viewing direction and the reference frame of a camerain accordance with the invention.

FIG. 7 is a schematic illustration of an imaging system where sound andmotion that is recreated when microphones and motion sensors are not atthe optical center of a panoramic device in accordance with theinvention.

DETAILED DESCRIPTION

The present invention combines panoramic visual images, directionalsound, and directional motion. Panoramic visual images can comprise oneor more individual still images, or a sequence of images such as a videostream. The present invention provides that sound recorded with at leastone sound capturing device and motion recorded with at least one motioncapturing device can be heard and felt in conjunction with visualpanoramic images recorded with a panoramic visual image recording deviceduring playback. Directional sound and directional motion can beassociated with a particular view of visual images during playback toconvey a realistic experience to the viewer.

As used herein, the term “panoramic visual images” means wide angleimages taken from a field of view of from greater than 180° to 360°,such as from about 270° to 360°. In a particular embodiment, the fieldof view is up to 360° around a principal axis, which is often orientatedto provide a 360° horizontal field of view. In this embodiment, asecondary axis may be defined, e.g., a vertical field of view. Such avertical field of view may typically range from 0.1° to 180°, forexample, from 1° to 160°. In accordance with the present invention,sections of the panoramic visual images may be selectively viewed. Forexample, while the panoramic visual images may comprise up to a 360°field of view, a smaller section may be selectively displayed, e.g., afield from about 1° to about 60° may be selectively viewed.

An embodiment of the present invention provides an imaging systemincluding a panoramic visual image display, an associated directionalsound playback device, and an associated directional motion reproductiondevice. As used herein, the term “associated directional sound” meanssound corresponding to a particular image view. As used herein, the term“associated directional motion” means motion corresponding to aparticular image view.

FIG. 1 is a schematic diagram illustrating a system 10 for producingpanoramic visual images. A mirror 12 having optical axis 14 gatherslight 16 from all directions and redirects it to a camera 18. A cameraand panoramic mirror arrangement as shown in FIG. 1, or any othersuitable panoramic imaging device, may be used to capture the panoramicvisual images.

FIG. 2 is a raw image from a panospheric camera. FIG. 3 is the imagefrom FIG. 2 displayed as a rectangular image using a projection onto acylindrical surface. A viewer can select any part of this image toexamine, e.g., as shown in the framed region. Each panoramic image maydisplay a full 360° view from a point or set of points. These images maybe produced by any suitable type of panoramic camera, and may be viewedusing any suitable projection, such as perspective, cylindrical and/orspherical, on a panoramic visual image display device such as televisionscreens or computer monitors. At any given time, only part of the imagemay be displayed to the user, based on commands given to the system.

Sound and motion may be recorded simultaneously with the panoramicvisual imagery. As shown in FIG. 4A, a panoramic visual image recordingdevice 20 may be coupled with two or more associated sound capturingdevices 22 and at least one associated motion capturing device 24. Thesound capturing devices 22 may be located adjacent to the panoramicvisual image recording device 20, and/or may be located remotely fromthe panoramic visual image recording device 20. Additional soundcapturing devices 26 may also be positioned remotely from the panoramicvisual image recording device 20. Likewise, the associated motioncapturing device 24 may be located adjacent the panoramic visual imagerecording device 20, or may be located remotely from the panoramicvisual image recording device 20. The audio recordings received from theassociated sound capturing devices 22 and/or 26 and the captured motiondata of the associated motion capturing devices 24 may be synchronizedwith panoramic images captured by the panoramic visual image recordingdevice 20.

As the panoramic visual image recording device 20 collects panoramicimages, sound waves 28 produced from various sources of sound 30, mayalso be captured by the associated sound capturing devices 22 and themotion captured by the associated motion capturing device(s) 24. Thelocation of the corresponding source of the sound and motion may becomputed. As the view of the panoramic visual image is changed by acomputer utilizing a pre-determined program, or by the user manipulatinga haptic device such as a joystick, the audio playback and motionsimulation reproduction is altered such that the user is able toperceive the direction of the source of sound and the direction of themotion.

An associated sound capturing device may be used to capture sound on twoor more channels, and a temporal and spatial correspondence may beestablished between the panoramic visual image and the associated sound.Example associated sound capturing devices include microphones and thelike.

Sound can be captured, for example, by any number of microphones each ofwhich might be substantially uni-directional or omni-directional. Asused herein, the term “uni-directional” means sound is captured within acone oriented substantially in a single direction. As used herein, theterm “omni-directional” means sound is captured from all directions.Omni-directional microphones may be approximated by the use of severaldirectional microphones placed in a ring. Sound may also be recordedseparately, and an artificial coupling may be made between the panoramicvisual image and the associated sound. In another embodiment, either thepanoramic visual image and/or the associated sound are synthetic. Inthis embodiment, artificial panoramas created by computer models can beused in place of a real panoramic visual image and artificial sources ofsound, such as computer generated sound, may be used. In anotherembodiment, a different sound that has been recorded separately may beassociated with the panoramic visual image. Sound may be captured by asingle sound capturing device or a series of sound capturing devicesplaced in an environment being captured by a panoramic visual imagerecording device, such as, for example, a moving automobile.

The sources of sound may be point sources, such as a singer on a stage,or a diffuse source, such as an applauding audience. The spatialcorrespondence between the panoramic images and sound can be achieved bylocalizing the sources of sound and embedding the localized informationin a data stream that contains both panoramic image data and sound data.The method for localizing the sources of sound may include measuringboth the loudness and phase of the sound. From these measurements, anestimate of the location of the sources can be computed. If thepanoramic images are generated using a rotating device, one rotatingsound capturing device can also be used to simulate two or morestationary sound capturing devices.

Likewise, an associated motion capturing device may be used to capturemotion, and a temporal and spatial correspondence may be establishedbetween the panoramic visual image and the associated motion. Examplemotion capturing devices include gyroscopic motion sensors and the like.The motion captured can include linear velocity, rotational velocity,acceleration, pitch, or yaw sensed from an observation point withrespect to a scene in the panoramic visual image. In one embodiment, themotion capturing device can comprise multiple gyroscopic motion sensors.In another embodiment, the motion capturing device can comprise a singlethree-dimensional gyroscopic motion sensor.

Motion can be captured, for example, by any number of motion capturingdevices each of which may be placed throughout an environment beingcaptured by a panoramic visual image recording device, such as in amoving automobile. The motion capturing device may be capable of sensingindependent motion in different areas of the environment. Motion mayalso be recorded separately, and an artificial coupling may be madebetween the panoramic visual image and the motion. In anotherembodiment, the panoramic visual image and/or the motion are synthetic.An artificial panoramic visual image created by a computer model can beused in place of a real panoramic visual image and an artificial sourceof motion, such as computer generated motion, can be used. In anotherembodiment, a different motion that has been recorded or reproducedseparately may be associated with the panoramic visual image.

Panoramic visual images, associated directional sound and associatedmotion captured as described above can be synchronously reproduced to aviewer. As shown in FIGS. 4B and 4C, various systems can be constructedto reproduce the panoramic visual images, associated sound andassociated motion. As shown in FIG. 4B, recorded panoramic visual images30, recorded associated sound 32 and recorded associated motion 34 aretransmitted to a processing unit 36, such as a computer, to transmit therecorded visual images, sound and motion to an appropriate playbackdevice. The recorded panoramic visual images 30 are transmitted to apanoramic visual image display 38 and displayed to a viewer. Examples ofpanoramic visual image displays 38 may include various types of computermonitors, televisions, video projection systems, head mounted displays,holograms and the like. As shown in FIG. 4C, the panoramic visual imagedisplay 138 can comprise a curved screen, such as a screen thatpartially or fully surrounds a viewer. The panoramic visual imagedisplay may comprise a single display device, or multiple displaydevices such as a row or array of devices. As shown in FIG. 4B, therecorded associated sound 32 is transmitted to an associated directionalsound playback device or devices 40 and projected to a viewer. Examplesof associated directional sound playback devices 40 may include one ormore speakers driven by any suitable power source such as one or moreamplifiers. As shown in FIG. 4C, associated directional sound playbackdevices 140 can be positioned at various locations around a viewer, suchas at four ordinal directions or mounted in association with a visualimage display 138. As shown in FIG. 4B, the recorded associated motion34 is transmitted to an associated directional motion reproductiondevice 42. Examples of associated directional motion reproductiondevices 42 may include platforms 43 or chairs 44 containing hydraulicand other mechanical actuators that may be controlled by a computer orother suitable device to simulate motion. These motion reproductiondevices 42 may be located in proximity to a panoramic visual imagedisplay 38 and/or an associated sound playback device or devices 40. Asshown in FIG. 4C, the associated motion reproduction device(s) 142 mayalso be an integral unit with a panoramic visual image display 138and/or one or more associated sound reproduction devices 140. In oneembodiment, a user may be physically located in or on the associatedmotion reproduction device 142, such as sitting in a chair or standingon a platform.

In one embodiment of the invention, panoramic image video, directionalsound and measurements of the dynamics of motion may all be recordedover time and synchronized to a common time code. In this embodiment, acommon time code may be generated by a device and stamped on the digitalsamples produced by the panoramic visual image recording device, theassociated sound capturing device and the associated motion capturingdevice. This information may be digitally encoded and stored on acomputer memory device, digital tape deck, or other suitable recordingmedia.

One possible system for capture of these samples will collect inputrespectively from the panoramic visual image capturing device, the soundcapturing device and the motion capturing device into a common recordingsystem. Data from each subsystem will be assembled into samples of afixed duration and delivered periodically to the recording system. Eachdata stream may maintain an independent sampling rate appropriate forthe media, such as, for example, 30 frames per second of imagery orvideo, 44,100 sound samples per second, and 100 samples per second ofmotion data. The sampling rate may be pre-determined for a givenrecording session and can be held constant while a recording isin-progress.

In one embodiment, each of the subsystems will be constrained to operatein “real-time”, meaning they capture their respective data and deliverit to the recording system on a real-time basis with a consistentlatency, or time offset between the moment of record and the moment thedata is received by the recording system. By maintaining apredetermined, calibrated, or calculated latency from each data stream,data received from each stream can be tagged with a time stamp. The timestamp can be determined by an internal or external clock mechanism thatis part of the recording system. A sample from a particular data streamcan be stamped with the current clock time as shown in Equation 1, wherethe time stamp out is equal to the time input minus the correspondingstream latency.T[out]=T[in]−L[stream]  Equation 1:

Accordingly, samples from each data stream may be synchronized with eachother within a particular error range. In a basic system, the maximumerror can correspond to the longest sampling interval of any individualdata stream, for example, 1/30th of a second. For bettersynchronization, the disparate recording systems can be triggered torecord their samples based on a common signaling device, commonly knownas a “genlock” in video production. This will allow arbitrarily smallerror ranges for captured samples.

When the recording is played back, the time code may be used toreassemble the various panoramic video, sound and motion elements intheir original order and similitude, resulting in an accurate model ofthe recorded environment. Recorded data can be played through a playbackcomputer system 36, for example as shown in FIG. 4B. The playback systemmay have output devices for a video stream, multi-channel audio andmotion control. The playback system transmits data to these outputdevices by transmitting samples over individual output streams, eachoperating in real-time at a pre-determined sampling rate with a knownlatency. As an image or video stream is played, the playback system issignaled by an internal or external clock offset to correspond to thetime stamps recorded in the data stream. The clock may progress in“real-time”, for example 1 second: 1 second rate, or it may progress atan arbitrary scale factor of this rate for slow motion or fast playbackpurposes. The clock may also be triggered momentarily at a given momentin time corresponding to a time in the recorded data set. This can beuseful for pause or search capabilities.

Regardless of the mode of operation, to play back at a given time indexthe playback system will determine the sample time to output as shown inEquation 2 where the time out is based on the clock time minus themaximum system latency plus the latency of the stream.T[out]=T[in]−L[max]+L[stream]  Equation 2:

The output system may also use a queue mechanism to schedule samples tobe output at a certain time in the future. The underlying operatingsystem of the playback computer offers scheduling primitives to executethe output of the stream at the indicated time. Using such real-timefacilities of the host system or media architecture are preferred toensure better performance and timely delivery of data.

In one embodiment, the recorded data stream can provide a self-containedsynchronized stream of panoramic video, sound and motion display, suchthat it is possible for the viewer(s) of such a system to have a degreeof control over their experience. Specifically, the viewer(s) maycontrol their viewing direction and field-of-view or zoom interactivelyby using an input device such as a mouse, trackball, joystick, orgyroscope. In response to such input, the synchronized data streams canadjust to the new viewing parameters dynamically, panning the videoalong with sound and adjusting the motion data appropriately.

In another embodiment, the digitally recorded panoramic image video,sound and motion data may be transmitted from devices capable ofrecording this information directly to a playback device, such as adriving simulator or a flight simulator. In another embodiment, thisdigitally recorded data may be temporarily stored in an intermediatestorage device prior to being sent to such a simulator device for thepurpose of creating a time delay.

FIG. 5 schematically illustrates how the loudness of a sound from aparticular sound source and the type of motion that is reproduceddepends upon its location with respect to the current viewing directionof the panoramic visual image. The sound source 50 and motion source 52appears to come slightly from the left given the viewing direction 54and the field of view 56 shown. As the viewing direction changes, thestrength of the sound and the various motion parameters reproduced, suchas forces and accelerations, can be made to vary as a function of theangle between the viewing direction and the direction to the source ofthe sound and the motion. For the apparent angle θ, which can be definedas the angle between the viewing direction and the direction to thesource of the sound and the motion, one such function may beproportional to cos(θ/2).

In FIG. 5, for example, if panoramic image video, sound and motion werebeing recorded in a moving automobile, sound source 50 and motion source52 might represent sounds and motion associated with the viewer lookingout of the front of the automobile, and second sound source 58 andsecond motion source 60 may be associated with the directional soundsand motion associated with the viewer looking out of the rear of theautomobile. In this example, if the viewer is looking out of the frontof the automobile, indicated in FIG. 5 as the current viewing direction,and the car is traveling uphill, sound source 50 may project loud soundsignals 62, such as sounds in front of the car, whereas sound source 58may project faint sound signals 64. Given the current viewing direction,motion source 52 may be indicative of the viewer being tilted backwardsas the car climbs the hill. Likewise, if the viewing direction werechanged so as to be pointing towards sound source 58 and motion source60, representing the viewer turned 180° and looking out of the back ofthe car, sound source 58 may become loud while sound source 50 willbecome faint, and motion source 60 would be indicative of the viewerbeing tilted downwards, indicating the car is still traveling uphill.

If the source of the sound and motion parameters, such as velocity ordirection, remains constant, the user may be able to sense the unvaryingdirection of the sound and motion as the viewing direction is changed.Alternatively, if the direction of the source of sound and motion ischanging, the user may be able to sense the direction of the movingsource as the selected view of the panoramic imagery is changed. One ormore sound reproduction devices may be used to play back the associatedsound and at least one motion reproducing device may be used toreproduce the associated motion.

If there is only one associated sound reproduction device, the loudnessof the sound may be modulated according to the alignment of the soundsource with respect to the current viewing angle. The viewing angle maybe chosen by a user, may be pre-determined, or may be automaticallyvaried as a user “looks” around a scene. If multiple associated soundreproduction devices are used for playing back the sound, the soundplayed back from the sound reproduction devices may be modulated so asto provide the listener with the feedback.

FIG. 6 is the rectangular projected image of FIG. 3, illustrating anangle between the viewing direction (the center of the selected view)and the reference frame of the camera. Each row of the image, such asrows of pixel data, spans 180°. If the viewing direction points to thelocation of the sound and motion source, then the sound will be at itsloudest and the motion from that particular direction will bereproduced. If the sound originates 180° from the viewing direction, thesound will be at its faintest and the motion from this new viewingdirection will be reproduced.

If there is more than one sound reproduction device, the phase andloudness of the sound on the sound reproduction devices may be modulatedto emulate the position of the sound source with respect to the currentviewing direction. Although there may be no depth information from thecamera, the amount of zoom selected by the user could be interpreted asa depth cue to select the sound balance between the two sound capturingdevices. Likewise, the amount of zoom selected by the user could beinterpreted as a cue to select the mix of motion reproduced between themotion capturing devices. The zoom could also be used to alter theloudness of the sound so as to correspond with the experience of gettingcloser or farther away from the source of the sound. Thus, sound andmotion may be recreated as coming from a direction without knowing itsexact position.

If the directional sound capturing devices and motion capturing devicesare not at the optical center of the panoramic visual image recordingdevice, using the angular difference between the viewing direction andthe sound capturing device direction and motion capturing devicedirection may not be sufficient. For example, three omni-directionalmicrophones 70, 72 and 74 and three gyroscopic motion sensors 76, 78 and80 can be placed in an environment as illustrated in FIG. 7. A distancecan be recreated by modulating the sound from the two microphones andmixing the appropriate amount of motion from the two gyroscopic motionsensors as a function of the angular difference between the viewingdirection 82 of the field of view 84 and the axis of the microphone andgyroscopic motion sensor base line. In FIG. 7, b₁₂ is the base linedistance between microphones 70 and 72 and gyroscopic motion sensors 76and 78, while b₂₃ is the distance between microphones 72 and 74 andgyroscopic motion sensors 78 and 80. The angle θ is the angle betweenthe baseline and the viewing direction 82.

Only those microphones and gyroscopic motion sensors that fall in thefield of view may be used to recreate the sound and motion. In theembodiment shown in FIG. 7, microphones 70 and 72 and gyroscopic motionsensors 76 and 78 are used while microphone 74 and gyroscopic motionsensor 80 are not used. If the field of view 84 were to rotateclockwise, microphone 70 and gyroscopic motion sensor 76 may not be usedbut microphones 72 and 74 and gyroscopic motion sensors 78 and 80 wouldbe used. The sound and motion is compiled based on combinations of thesound recorded at two microphones and the motion captured at twogyroscopic motion sensors. This may be determined for every pair ofmicrophones and every pair of gyroscopic motion sensors in the viewingarea.

The strengths of the sounds from the microphones and the recorded motionfrom the gyroscopic motion sensors may be combined as follows. Therelative strength of the signal of microphone i is determined byEquation 3,(b _(ij) −d _(i))/b _(ij)  Equation 3:

-   -   where b_(ij) is the baseline distance between microphones i and        j and d_(i) is the distance between the microphone i and the        intersection of the axis of the viewing direction and b_(ij).        The effect of direction (the offset θ) may be computed as        illustrated in FIG. 5. The magnitudes of the various motion        parameters captured by the gyroscopic motion sensors may also be        combined in a similar manner.

In one embodiment of the invention, one panoramic camera may be used inconjunction with multiple microphones and motion capturing devices. Themicrophones may have directionality, i.e., they are sensitive to soundscoming from the direction that they are pointed, with sensitivityfalling off in other directions, and the motion capturing devices mayalso have directionality, i.e., they are sensitive to motion coming fromthe particular location they are placed, with the sensitivity fallingoff in other directions. The microphones and motion capturing devicesmay have overlapping fields of sensitivity. Any sound and/or motion inthe environment may be detected by at least two microphones and by atleast two motion capturing devices.

In another embodiment, any sound and/or motion in the environment may bedetected by at least two microphones and a single three-dimensionalmotion capturing device. Sound and motion from the environment, sensedfrom an observation point in the scene, is recorded simultaneously withthe panoramic image video and may be correlated to the video for directtransmission or playback. The camera may have a natural frame ofreference and sounds and motion parameters may be located either byposition or direction (or both) with respect to the frame of reference.When the panoramic images are unwarped, the direction that the viewerchooses defines the offset from the camera reference frame. This offsetmay change dynamically with the selected view. The signal recorded fromeach sound capturing device and sensed and captured from each motioncapturing device may be played back in a modified manner based on theoffset from the panoramic visual image capturing device reference frame.For example, sound from each microphone and motion from each gyroscopicsensor may be combined depending on the total number of respectiveplayback devices. This may be particularly important for audioreproduction, e.g., if only one speaker is available, then the soundsrecorded from all microphones may be simply added up.

If the offset between the direction of the microphone i and the camerareference is denoted by θ_(i) the strength of the signal associated withthat microphone, M_(i), is cos(θ_(i))+ε, where ε is a minimal level ofsound playback. The composite sound is created determined by Equation 4:Σ(cos(θ_(i))+ε)·M _(i)  Equation 4:

If the playback device consists of multiple speakers, the sound may bedistributed to each speaker such that each speaker only plays the soundscorresponding to microphones pointed in a certain sector. For example,if four speakers are used, each speaker may only play sounds attributedto microphones in a 90° sector.

If the offset between the direction of the motion capturing device j inthe camera reference is denoted by θ_(j), the strength of the signalassociated with that motion capturing device, D_(j), is cos(θ_(j))+α,where α is a minimal level of motion reproduction. The composite motionis created according to Equation 5:Σ(cos(θ_(j))+α)·D _(j)  Equation 5:

If the motion reproduction device includes multiple parts, such asmultiple hydraulic or other mechanical actuators all attached to aplatform or a chair, the motion may be distributed to each hydraulicactuator such that each actuator only reproduces the motioncorresponding to a motion capturing device located in a certain sector.For example, if four hydraulic actuators are used, each actuator mayonly react to motion attributed to one or more motion capturing devicesin a 90° sector.

For playback of data streams involving motion control, it may beundesirable to allow arbitrary motion paths to be executed on the motioncontrol device. “Arbitrary paths” may be considered raw motion capturestreams that have not been filtered to within safe tolerances, or it maybe related to playing back sample data at a rate other than 1.0. Forinstance, pausing a stream at an excessive incline and playing back atfast rates that could induce unacceptable rates of acceleration maypresent dangerous conditions for the viewer(s). For these reasons,recorded motion can be filtered through a safety protocol.

The safety protocol ensures that motion control instructions do notexceed pre-defined safety limits for the safety of participants. Theserequirements can be specified for each output motion axis as a maximumrange of motion, maximum velocity, and/or maximum acceleration. When acontrol signal meets or exceeds any of these maximum values, the signalcan be modified to comply with the acceptable safety limits. This may beaccomplished through truncation (a stop or slowing motion before themaximum value is exceeded), attenuation (dampening or scaling motionsuch that the total range of motion over time never exceeds a maximumlimit), or override (the issuing of false motion or rejecting commandsto ensure smooth continuity of motion). In one embodiment, the safetyprotocol can run in two passes. A first pre-playback pass can scan theentire set of motion control samples in a recording, and perform thenecessary filtering to ensure compliance of the stream. This allows formotion more consistent with the recorded intent. The second pre-issuepass will filter motion control statements before they are executed bythe motion control device. In theory, no signals should be issued thatexceed safety protocols because of the first pass, however, the secondpass ensures a final protection in case of first pass failure orsignaling errors.

The nature of the panoramic video, sound and motion capture devicesalong with the synchronized recording system makes it possible for liveplayback of a data stream. This means that as an event is recorded inone area, it can be encoded and transmitted to a remote playback devicefor immediate playback. For example, capture devices mounted inside of aracecar vehicle can transmit data live to a playback system locatedelsewhere. Participants can ride as a passenger as a racecar is drivenat another location. The safety protocols will ensure motion capturedata will be attenuated to safe levels.

Whereas particular embodiments of this invention have been describedabove for purposes of illustration, it will be evident to those skilledin the art that numerous variations of the details of the presentinvention may be made without departing from the invention as defined.

1. An apparatus comprising: a panoramic visual image capturing devicecapable of recording panoramic visual images; a sound capturing devicecapable of recording sound associated with viewing directions for thevisual images; and a motion capturing device capable of recording motionassociated with viewing directions for the visual images.
 2. Theapparatus of claim 1, further comprising at least one additional soundcapturing device positioned remotely from the panoramic visual imagecapturing device.
 3. The apparatus of claim 1, further comprising atleast one additional motion capturing device positioned remotely fromthe panoramic visual image capturing device.
 4. The apparatus of claim1, further comprising a processor for processing signals from the soundcapturing device and the motion capturing device to determine a locationof a source of sound and motion.
 5. A method comprising the steps of:recording panoramic visual images; recording sound associated withviewing directions for the visual images; and recording motionassociated with viewing directions for the visual images.
 6. The methodof claim 5, wherein the sound associated with viewing directions for thevisual images is recorded at a position remote from a visual imagecapturing device.
 7. The method of claim 5, wherein the motionassociated with viewing directions for the visual images is recorded ata position remote from a visual image capturing device.
 8. The method ofclaim 5, wherein the sound associated with viewing directions for thevisual images is recorded on two or more channels, and a temporal andspatial correspondence is established between the panoramic visual imageand the recorded sound.
 9. The method of claim 5, wherein at least oneof the panoramic visual images, the recorded sound and/or the recordedmotion is synthetic.
 10. The method of claim 5, wherein a spatialcorrespondence between the panoramic images and recorded sound isachieved by localizing sources of the sound and embedding localizedinformation in a data stream that contains both panoramic image data andsound data.
 11. The method of claim 10, wherein the sources of sound arelocalized by measuring both the loudness and phase of the sound, and bycomputing an estimate of the location of the sources.
 12. The method ofclaim 5, wherein the step of recording motion associated with viewingdirections for the visual images includes the steps of: sensing motionin different areas of an environment; and coupling the sensed motion tothe panoramic visual images.
 13. The method of claim 5, wherein thesound and motion are recorded over time and synchronized to a commontime code.
 14. The method of claim 13, wherein the common time code isgenerated and stamped on digital samples of the panoramic visual image,the recorded sound, and the recorded motion.
 15. An image playbacksystem comprising: a panoramic visual image display for displayingpanoramic visual images; a sound playback device for reproducing soundassociated with viewing directions for the panoramic visual images; anda motion reproduction device for reproducing motion associated withviewing directions for the panoramic visual images, wherein thepanoramic visual images, sound and motion are synchronously reproduced.16. An image playback method comprising the steps of: displayingpanoramic visual images; reproducing sound associated with viewingdirections for the panoramic visual images; and reproducing motionassociated with viewing directions for the panoramic visual images,wherein the panoramic visual images, sound and motion are synchronouslyreproduced.
 17. The image playback method of claim 16, furthercomprising the step of: changing the viewing directions for thepanoramic visual image by utilizing a pre-determined computer program,or by a user manipulating a haptic device.
 18. The image playback methodof claim 16, wherein as the viewing direction changes, the strength ofthe sound and the motion parameters vary as a function of the anglebetween the viewing direction and a direction to a source of the soundand motion.
 19. The image playback method of claim 16, wherein the soundis modulated according to the alignment of a sound source with respectto a current viewing direction.
 20. The image playback method of claim16, wherein the phase and loudness of the sound is modulated to emulatea position of a sound source with respect to a current viewingdirection.
 21. The image playback method of claim 16, wherein a userchooses a direction of view that defines an offset from a referenceframe, and the sound and motion are played back in a modified mannerbased on the offset from the panoramic visual image reference frame.